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IN THE CLAIMS 

The following listing of claims is provided in accordance with 37 C.F.R. §1.121: 

1 . (currently amended) A system for assessing the performance of an 
algorithm during development, comprising: 

a design of experiments component that establishes an acceptable number of 
experiments for analyzing the algorithm; 

an experiment performance component that runs the established number of 
experiments for the algorith m, wherein the experiment performance component 
comprises a performance metric component that evaluates the results of the experiments 
run for the algorithm, the evaluation comprising comparing the results with a baseline 
algorithm and a performance metric ; and 

a simulation component that simulates the behavior of the algorithm using results 
from the experiment performance component. 

2. (original) The system according to claim 1, wherein the design of 
experiments component selects an appropriate level of experiments for analyzing the 
algorithm. 

3. (original) The system according to claim 2 5 wherein the design of 
experiments component selects the appropriate level of experiments from at least one of 
full factorial experiments, fractional factorial experiments or screening experiments. 

4. (cancelled). 

5. (currently amended) The system according to claim [[4]] 1, wherein the 
experiment performance component comprises an algorithm adjustment component that 
adjusts logic or parameters of the algorithm for unacceptable results. 
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6. (original) The system according to claim 5, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
after adjustment of logic or parameters. 

7. (original) The system according to claim 1, further comprising a 
module insertion component that inserts a module into the algorithm. 

8. (original) The system according to claim 7, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
including the inserted module. 

9. (original) The system according to claim 1, wherein the simulation 
component performs a Monte Carlo simulation. 

10. (original) The system according to claim 1, wherein the simulation 
component uses at least one confusion matrix to simulate the behavior of the algorithm. 

11. (original) The system according to claim 1, further comprising a 
simulation performance component that evaluates the performance of the simulation of 
the algorithm. 

12. (currently amended) The system according to claim 11, wherein the 
simulation performance component comprises an algorithm adjustment component that 
adjusts logic or parameters of the algorithm for unacceptable results , wherein the 
adjustment is performed during the development of the algorithm . 

13. (original) The system according to claim 12, wherein the simulation 
component runs the simulation for the algorithm after adjustment of logic or parameters. 
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14. (currently amended) A system for assessing the performance of an 
algorithm during development, comprising: 

a design of experiments component that establishes an acceptable number of 
experiments for analyzing the algorithm; 

an experiment performance component that runs the established number of 
experiments for the algorithm , wherein the experiment performance component 
comprises a performance metric component that evaluates the results of the experiments 
run for the algorithm, the evaluation comprising comparing the results with a baseline 
algorithm and a performance metric ; 

a simulation component that simulates the behavior of the algorithm using results 
from the experiment performance component; and 

a simulation performance component that evaluates the performance of the 
simulation for the algorithm. 

15. (original) The system according to claim 14, wherein the design of 
experiments component selects an appropriate level of experiments for analyzing the 
algorithm. 

16. (original) The system according to claim 15, wherein the design of 
experiments component selects the appropriate level of experiments from at least one of 
full factorial experiments, fractional factorial experiments or screening experiments. 

17. (cancelled). 

18. (currently amended) The system according to claim [[17]] 14, wherein 
the experiment performance component comprises an algorithm adjustment component 
that adjusts logic or parameters of the algorithm for unacceptable results. 
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19. (original) The system according to claim 18, wherein the experiment 
performance component runs the established number of experiments for the portion of the 
algorithm after adjustment of parameters. 

20. (original) The system according to claim 14, further comprising a 
module insertion component that inserts a module into the algorithm. 

21. (original) The system according to claim 20, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
including the inserted module. 

22. (original) The system according to claim 14, wherein the simulation 
component performs a Monte Carlo simulation. 

23. (original) The system according to claim 14, wherein the simulation 
component uses at least one confusion matrix to simulate the behavior of the algorithm. 

24. (currently amended) The system according to claim 14, wherein the 
simulation performance component comprises an algorithm adjustment componentthat 
adjusts logic or parameters of the algorithm for unacceptable results , wherein the 
adjustment is performed during the development of the algorithm . 

25. (original) The system according to claim 24, wherein the simulation 
componentruns the simulation for the algorithm after adjustment of logic or parameters. 

26. (currently amended) A system for assessing the performance of an 
algorithm during development, comprising: 

a design of experiments componentthat establishes an acceptable number of 
experiments for analyzing the algorithm; 
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an experiment performance component that runs the established number of 
experiments for the algorithm and uses a performance metric to evaluate the results of the 
experiments , the evaluation comprising comparing the results with a baseline algorithm 
and the performance metric; 

a Monte Carlo simulation component that simulates the behavior of the algorithm 
using results from the experiment performance component with a Monte Carlo 
simulation; and 

a simulation performance component that evaluates the performance of the Monte 
Carlo simulation for the algorithm. 

27. (original) The system according to claim 26, wherein the design of 
experiments component selects an appropriate level of experiments from at least one of 
full factorial experiments, fractional factorial experiments or screening experiments. 

28. (original) The system according to claim 26, wherein the experiment 
performance component comprises an algorithm adjustment component that adjusts logic 
or parameters of the algorithm for unacceptable results. 

29. (original) The system according to claim 28, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
after adjustment of logic or parameters. 

30. (original) The system according to claim 26, wherein the Monte Carlo 
simulation component uses at least one confusion matrix to simulate the behavior of the 
algorithm. 
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31. (currently amended) The system according to claim 26, wherein the 
simulation performance component comprises an algorithm adjustment component that 
adjusts logic or parameters of the algorithm for unacceptable results , wherein the 
adjustment is performed during the development of the algorithm . 

32. (original) The system according to claim 31, wherein the simulation 
component runs the simulation for the algorithm after adjustment of parameters. 

33. (currently amended) A system for assessing the performance of an 
algorithm during development, comprising: 

a design of experiments component that establishes an acceptable number of 
experiments for analyzing the algorithm; 

an experiment performance component that runs the established number of 
experiments for the algorithm; 

a performance metric component that evaluates the results of the experiments run 
for the algorithm , the evaluation comprising comparing the results with a baseline 
algorithm and a performance metric ; and 

an algorithm adjustment component that adjusts logic or parameters of the 
algorithm for unacceptable results , wherein the adjustment is performed during the 
development of the algorithm . 

34. (original) The system according to claim 33, wherein the design of 
experiments component selects an appropriate level of experiments for analyzing the 
algorithm from at least one of full factorial experiments, fractional factorial experiments 
or screening experiments. 

35. (original) The system according to claim 33, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
after adjustment of logic or parameters. 
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36. (original) The system according to claim 33, further comprising a 
module insertion component that inserts a module into the algorithm. 

37. (original) The system according to claim 36, wherein the experiment 
performance component runs the established number of experiments for the algorithm 
including the inserted module. 

38. (currently amended) A system for assessing the performance of an 
algorithm during development, comprising: 

a Monte Carlo simulation component that simulates the behavior of the algorithm 
with a Monte Carlo simulation, wherein the Monte Carlo simulation component uses at 
least one confusion matrix to simulate the behavior of the algorithm, wherein the Monte 
Carlo simulation component further computes a z-score based on one or more entries in 
the confusion matrix, and wherein the z-scores generate random values that reflect the 
behavior of the algorithm in the confusion matrix; 

a simulation performance component that evaluates the performance of the Monte 
Carlo simulation for the algorithm; and 

an algorithm adjustment component that adjusts logic or parameters of the 
algorithm for unacceptable results , wherein the adjustment is performed during the 
development of the algorithm . 

39. (original) The system according to claim 38, wherein the Monte Carlo 
simulation component runs the simulation for the algorithm after adjustment of logic or 
parameters. 



40.-76. (canceled). 
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77. (currently amended) A computer-readable medium storing computer 
instructions for instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 

using design of experiments to establish an acceptable number of experiments for 
analyzing the algorithm; 

running the established number of experiments for the algorithm, using a 
performance metric to evaluate the results of the experiments run for the algorithm, 
wherein the evaluation comprises comparing the results with a baseline algorithm and the 
performance metric ; and 

simulating the behavior of the algorithm using results from the experiments. 

78. (original) The computer-readable medium according to claim 77, 
wherein the using of design of experiments comprises instructions for selecting an 
appropriate level of experiments for analyzing the algorithm. 

79. (original) The computer-readable medium according to claim 78, 
wherein the appropriate level of experiments comprise at least one of full factorial 
experiments, fractional factorial experiments or screening experiments. 

80. (cancelled). 

8 1 . (currently amended) The computer-readable medium according to claim 
[[80]] 77, further comprising instructions for adjusting logic or parameters of the 
algorithm for unacceptable results. 

82. (original) The computer-readable medium according to claim 81, 
further comprising instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 
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83. (original) The computer-readable medium according to claim 77, 
further comprising instructions for inserting a module into the algorithm. 

84. (original) The computer-readable medium according to claim 83, 
further comprising instructions for running the established number of experiments for the 
algorithm including the inserted module. 

85. (original) The computer-readable medium according to claim 77, 
wherein the simulating comprises instructions for using at least one confusion matrix to 
simulate the behavior of the algorithm. 

86. (original) The computer-readable medium according to claim 77, 
further comprising instructions for evaluating the performance of the simulation for the 
algorithm. 

87. (currently amended) The computer-readable medium according to claim 
86, further comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results , wherein the adjustment is performed during the development of the 
algorithm . 

88. (original) The computer-readable medium according to claim 87, 
further comprising instructions for running the simulation for the algorithm after 
adjustment of logic or parameters. 

89. (currently amended) A computer-readable medium storing computer 
instructions for instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 

using design of experiments to establish an acceptable number of experiments for 
analyzing the algorithm; 
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running the established number of experiments for the algorithm, using a 
performance metric to evaluate the results of the experiments run for the algorithm, 
wherein the evaluation comprises comparing the results with a baseline algorithm and the 
performance metric ; 

simulating the behavior of the algorithm using results from the experiments; and 
evaluating the performance of the simulation for the algorithm. 

90. (original) The computer-readable medium according to claim 89, 
wherein the using of design of experiments comprises instructions for selecting an 
appropriate level of experiments for analyzing the algorithm from at least one of full 
factorial experiments, fractional factorial experiments or screening experiments. 

91. (cancelled). 

92. (currently amended) The computer-readable medium according to claim 
[[91]] 89, further comprising instructions for adjusting logic or parameters of the 
algorithm for unacceptable results. 

93. (original) The computer-readable medium according to claim 92, 
further comprising instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 

94. (original) The computer-readable medium according to claim 89, 
further comprising instructions for inserting a module into the algorithm. 

95. (original) The computer-readable medium according to claim 94, 
further comprising instructions for running the established number of experiments for the 
algorithm including the inserted module. 
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96. (original) The computer-readable medium according to claim 89, 
wherein the simulation is a Monte Carlo simulation. 

97. (original) The computer-readable medium according to claim 89, 
wherein the simulating uses at least one confusion matrix to simulate the behavior of the 
algorithm. 

98. (currently amended) The computer-readable medium according to claim 
89, further comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results , wherein the adjustment is performed during the development of the 
algorithm . 

99. (original) The computer-readable medium according to claim 98, 
further comprising instructions for running the simulation for the algorithm after 
adjustment of logic or parameters. 

100. (currently amended) A computer-readable medium storing computer 
instructions for instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 

using a design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm; 

using a performance metric to evaluate the results of the experiments run for the 
algorithm, wherein the evaluation comprises comparing the results with a baseline 
algorithm and the performance metric ; 

using a Monte Carlo simulation to simulate the behavior of the algorithm using 
results from the performance metric; and 

evaluating the performance of the Monte Carlo simulation for the he algorithm. 
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101. (original) The computer-readable medium according to claim 100, 
wherein using the design of experiments comprises instructions for selecting an 
appropriate level of experiments from at least one of full factorial experiments, fractional 
factorial experiments or screening experiments. 

102. (original) The computer-readable medium according to claim 100, 
further comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results. 

103. (original) The computer-readable medium according to claim 102, 
further comprising instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 

104. (original) The computer-readable medium according to claim 100, 
wherein the simulating comprises instructions for using at least one confusion matrix to 
simulate the behavior of the algorithm. 

105. (currently amended) The computer-readable medium according to claim 
100, further comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results , wherein the adjustment is performed during the development of the 
algorithm . 

106. (original) The computer-readable medium according to claim 105, 
further comprising instructions for running the simulation for the algorithm after 
adjustment of logic or parameters. 

107. (currently amended) A computer-readable medium storing computer 
instructions for instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 
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using a design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm, using a 
performance metric to evaluate the results of the experiments run for the algorithm, 
wherein the evaluation comprises comparing the results with a baseline algorithm and the 
performance metric ; 

evaluating the results of the experiments run for the algorithm with a performance 
metric; and 

adjusting logic or parameters of the algorithm for unacceptable results , wherein 
the adjustment is performed during the development of the algorithm . 

108. (original) The computer-readable medium according to claim 107, 
wherein the using of design of experiments comprises instructions for selecting an 
appropriate level of experiments for analyzing the algorithm from at least one of full 
factorial experiments, fractional factorial experiments or screening experiments. 

109. (original) The computer-readable medium according to claim 107, 
further comprising instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 

110. (original) The computer-readable medium according to claim 107, 
further comprising instructions for inserting a module into the algorithm. 

111. (original) The computer-readable medium according to claim 110, 
further comprising instructions for running the established number of experiments for the 
algorithm including the inserted module. 
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112. (currently amended) A computer-readable medium storing computer 
instructions for instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 

simulating the behavior of the algorithm with a Monte Carlo simulation, wherein 
the Monte Carlo simulation uses at least one confusion matrix to simulate the behavior of 
the algorithm , wherein the Monte Carlo simulation further computes a z-score based on 
one or more entries in the confusion matrix, and wherein the z-scores generate random 
values that reflect the behavior of the algorithm in the confusion matrix; 

evaluating the performance of the Monte Carlo simulation for the algorithm; and 
adjusting logic or parameters of the algorithm for unacceptable results , wherein 
the adjustment is performed during the development of the algorithm . 

113. (original) The computer-readable medium according to claim 112, 
further comprising instructions for running the Monte Carlo simulation for the algorithm 
after adjustment of logic or parameters. 



